Binaural cochlear implant processing

ABSTRACT

A sound processing arrangement is described for a patient with a bilateral cochlear implant system having implanted electrode arrays in each ear. There is a left-side sensing microphone and a right-side sensing microphone, each configured for sensing the sound environment surrounding the patient and generating corresponding microphone signals. A sound object identification module is configured for analyzing the microphone signals to identify one or more sound objects within the sound environment. A sound object selection module is configured for processing the microphone signals to generate a sound object signal for each of the one or more sound objects. A stimulation side selector module is configured for selecting on which side or sides of the bilateral cochlear implant arrangement to process each sound object signal. One or more sound processors processes the sound object signals to generate stimulation signals to the implanted electrode arrays on the selected side or sides.

This application is a continuation of U.S. patent application Ser. No.15/005,046, filed Jan. 25, 2016, which in turn is a continuation-in-partof Patent Cooperation Treaty Application PCT/US2014/047118, filed Jul.18, 2014, which in turn claims priority from U.S. Provisional PatentApplication 61/857,756, filed Jul. 24, 2014, each of which areincorporated herein by reference in their entireties.

FIELD OF THE INVENTION

The present invention relates to audio signal processing in cochlearimplant systems.

BACKGROUND ART

A normal ear transmits sounds as shown in FIG. 1 through the outer ear101 to the tympanic membrane (eardrum) 102, which moves the bones of themiddle ear 103 (malleus, incus, and stapes) that vibrate the oval windowand round window openings of the cochlea 104. The cochlea 104 is a longnarrow duct wound spirally about its axis for approximately two and ahalf turns. It includes an upper channel known as the scala vestibuliand a lower channel known as the scala tympani, which are connected bythe cochlear duct. The cochlea 104 forms an upright spiraling cone witha center called the modiolar where the spiral ganglion cells of theacoustic nerve 113 reside. In response to received sounds transmitted bythe middle ear 103, the fluid-filled cochlea 104 functions as atransducer to generate electric pulses which are transmitted to thecochlear nerve 113, and ultimately to the brain.

Hearing is impaired when there are problems in the ability to transduceexternal sounds into meaningful action potentials along the neuralsubstrate of the cochlea 104. To improve impaired hearing, auditoryprostheses have been developed. For example, when the impairment isrelated to operation of the middle ear 103, a conventional hearing aidmay be used to provide acoustic-mechanical stimulation to the auditorysystem in the form of amplified sound. Or when the impairment isassociated with the cochlea 104, a cochlear implant with an implantedstimulation electrode can electrically stimulate auditory nerve tissuewith small currents delivered by multiple electrode contacts distributedalong the electrode.

FIG. 1 also shows some components of a typical cochlear implant systemwhich includes an external microphone that provides an audio signalinput to an external signal processor 111 where various signalprocessing schemes can be implemented. The processed signal is thenconverted into a digital data format, such as a sequence of data frames,for transmission into the implant processor 108. Besides receiving theprocessed audio information, the implant processor 108 also performsadditional signal processing such as error correction, pulse formation,etc., and produces a stimulation pattern (based on the extracted audioinformation) that is sent through an electrode lead 109 to an implantedelectrode array 110. Typically, this electrode array 110 includesmultiple electrode contacts 112 on its surface that provide selectivestimulation of the cochlea 104.

It once was commonly the case that cochlear implant systems wereunilateral systems with only one ear being implanted with an electrodearray that delivers electrical stimulation signals to the implanted ear.More commonly today, cochlear implant systems often are bilateral withboth ears receiving implanted electrode arrays that deliver stimulationsignals to the implanted ears.

The human auditory processing system segregates specific sound objectsfrom complex auditory scenes using several binaural cues such asinteraural time and level differences (ITD/ILD) and monaural cues suchas harmonicity or common onset. This process is known as auditory sceneanalysis (ASA) as described more fully in A. S. Bregman Auditory SceneAnalysis: The Perceptual Organization of Sound, MIT Press, Cambridge,Mass. (1990), incorporated herein by reference. Hearing impairedpatients have difficulties successfully performing such an auditoryscene analysis even with a hearing prosthesis such as a cochlearimplant. Because of such problems, cochlear implant users often struggleto listen to a single individual sound source within a mixture ofmultiple sound sources as in a noisy sound environment. In the case ofunderstanding speech, this translates into reduced speechintelligibility. In the case of music, musical perception is degradeddue to the inability to successfully isolate and follow individualinstruments.

U.S. Patent Publication 20100135500 describes a binaural hearing systemwith microphones on either side of the patient's head based on comparingthe relative signal-to-noise ratios from each microphone. But there isno suggestion as to analysis and processing of sound objects in thesurrounding sound environment.

WO 2013/101088 by Mishra stated that in prior art systems the sensedipsilateral and contralateral signals were “compared as a whole andselect one of them for presentation to the patient based on thecomparison.” Mishra proposed to compare the ipsilateral andcontralateral signals on a channel-by-channel comparison, selectivelyamplifying the corresponding ipsilateral and contralateral signals(FIGS. 4 and 6) and then finally mixing the modified channel signalswhich was forwarded to the implanted cochlear implant. This approachdoes not consider components of an audio signal which may be correspondto the same sound object, e.g. the fundamental and first and/or secondharmonic may be treated in different ways (different gains) and thus thesound object may be distorted.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed to a sound processingarrangement for a patient with a bilateral cochlear implant systemhaving implanted electrode arrays in each ear. There is a left-sidesensing microphone and a right-side sensing microphone, each configuredfor sensing the sound environment surrounding the patient and generatingcorresponding microphone signal outputs. A sound object identificationmodule is configured for analyzing the microphone signals to identifyone or more sound objects within the sound environment. A sound objectselection module is configured for processing the microphone signals togenerate a sound object signal for each of the one or more soundobjects. A stimulation side selector module is configured for selectingon which side or sides of the bilateral cochlear implant arrangement toprocess each sound object signal. One or more sound processors processesthe sound object signals to generate stimulation signals to theimplanted electrode arrays on the selected side or sides.

In a specific embodiment, the one or more sound processors may processthe sound object signals based on adjusting a phase component and/or anamplitude component of each sound object signal. The stimulation sideselector module may be configured for using sound object time and/oramplitude difference components in the microphone signals to select onwhich side or sides of the bilateral cochlear implant arrangement toprocess each sound object signal. The sensing microphones may be locatednext to the ear on each side of the patient's head, or in the ear canalon each side of the patient's head.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows elements of a human ear and cochlear implant system.

FIG. 2 shows various functional blocks in a sound processing arrangementfor a unilateral cochlear implant system according to one embodiment ofthe present invention.

FIG. 3 shows an example situation for a single sound object which iscloser to the ipsilateral microphone.

FIG. 4 shows an example situation for two sound objects.

FIG. 5 shows various functional blocks in a sound processing arrangementfor a bilateral cochlear implant system according to one embodiment ofthe present invention.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Embodiments of the present invention are directed to sound processingarrangements and methods for a listener with a cochlear implant systemthat performs real time identification, selection and processing ofsound objects in the surrounding sound environment. Examples of suchsound objects include voices of individuals, musical instruments, andmore generally, any noise generating objects such as cars, etc. Soundobjects comprise several (or all) characteristic frequency features of aspecific sound source such as the fundamental frequency and higherharmonics, or a specific frequency characteristic. In general, soundobject are sets of complex sounds coming from a single exact position ata certain time and having specific frequency characteristics. Theinventive approach identifies sound objects in both left-side andright-side microphone signals, and thereby does not treat the entiresignals as a whole, but rather the individual sound objects themselvesare treated as a whole, and entire sound objects are mixed. Thisproduces binaural sound processing in unilateral and bilateral cochlearimplant systems with more accurate timing and level information for thesound objects, thereby providing improved localization and betterhearing of sound events.

FIG. 2 shows functional blocks in an embodiment for a unilateralcochlear implant system where a left-side microphone 201 and aright-side microphone 202, are each configured for sensing the soundenvironment surrounding the patient and generating correspondingmicrophone signals. The left-side and right-side microphones 201 and 202may be located next to the ear or in the ear canal on each side of thepatient's head. Typically, each microphone signal may be initiallyprocessed by one or more preprocessor modules 203 to initially analyzeand adjust the microphone signals.

A sound object identification module 204 is configured for analyzing themicrophone signals in real time together with an analysis of theacoustic properties of the sound environment to identify the individualsound objects (SO) that are present within the sound environment. Foreach k^(th) identified sound object, the sound object identificationmodule 204 calculates two sound object subsets, SO_(ki) and SO_(kc).

A sound object selection module 205 is configured for processing themicrophone signals to generate a sound object signal for each of the oneor more sound objects. For example, the sound object selection module205 may use sound object time difference components (i.e., phasedifference) and/or sound object amplitude components in the microphonesignals to select the sensing microphone closer to a given sound objectto enhance its microphone signal to produce the corresponding soundobject signal. If both microphone signals are substantially identical,the microphone closer to a sound object will provide a stronger signaland that signal will arrive earlier, and in this way, the sound objectselection module 205 can select the stronger and earlier microphonesignal for each sound object for processing to generate thecorresponding sound object signal. For each sound object, the soundobject selection module 205 outputs only the selected sound objectsignal. If the sound object selection module 205 does not identify anysound objects as present, then the sound may be processed from onemicrophone only, preferably from the left-side (ipsilateral) sensingmicrophone 201.

FIG. 3 shows an example situation where a given sound object (SO₁) isidentified to be closer to the ipsilateral left-side sensing microphone.The sound object selection module 205 in FIG. 2 will select only theleft-side microphone signal for use as the sound object signal. FIG. 4shows another situation with two different sound objects, one of which(SO₁) is closer to the left-side side and the other of which (SO₂) iscloser to the right-side side. The sound object selection module 205selects only SO_(1i) and SO_(2c) as sound object signals for output.

A sound processor 206 includes a sound object summation module 207 thatcombines the sound object signals from the sound object selection module205 and a stimulation signal processor 208 that generates one or morestimulation signals to the cochlear implant based on user-specificfitting characteristics. The sound processor 206 may combine the soundobject signals based on adjusting a phase component and/or an amplitudecomponent of each sound object signal.

The entire system operates in real time so as to correctly track movingsound objects. If the SO₁ in FIG. 3 were to be moving from left toright, the SO_(1c) microphone signal will be selected as the SO₁ soundobject signal as soon as SO₁ is identified to be closer to theright-side microphone 202. The system components may be encased in aprocessor housing that may be worn either on the body (e.g., behind theear) or that may be fully implantable.

The foregoing discussion relates to a unilateral cochlear implant systemwhere the recipient patient has just one implanted ear. Similarapproaches can also be applied in bilateral systems where both ears areimplanted. Basically, as in a unilateral system, the sound objects in abilateral system are analyzed in real time (in one or more signalprocessor), but the selected individual sound objects are only presentedto the ear which is closer to the sound object.

FIG. 5 shows various functional blocks in a sound processing arrangementfor a bilateral cochlear implant system according to one embodiment ofthe present invention, where blocks 201-205 are as in the unilateralembodiment in FIG. 2. And the sound object processing blocks 206/208 areeach implanted for each ear, but otherwise are the same as in FIG. 2.The two cochlear implants and their respective processing blocks arecommunicatively connected (wired or wireless) to exchange information asdescribed below. In addition to the blocks shown and described withrespect to FIG. 2, the system in FIG. 5 also includes a stimulation sideselection module 501 that receives the sound object signals from thesound object selection module 205 and selects which side or sides to usefor stimulation. Of course, modules 204, 205 and 501 can also becombined into a single physical module. If the selected subset with thestronger and earlier phase is obtained from the left-side sensingmicrophone 201, then only the left-side cochlear implant is selected forstimulation, and that sound object information is not provided to thecochlear implant on the right-side side, and vice versa. So in FIG. 4,sound object signal SO_(1i) would only be presented to one of thecochlear implants, and sound object signal SO_(2c) only to the otherone.

In a specific implementation for bilateral cochlear implant, the maximuminteraural time difference phase (or time) delay (ITD) between the twoears should be taken into account. This ITD typically may beapproximately 0.66 ms (considering the speed of the sound at sea level).Thus the length of the time window used for comparison of the left- andright-side sampled signal should be larger than 0.66 ms. The sampledamplitudes from both time windows of the sensing microphones can becompared. Then in order to select corresponding time windows of bothsides which have a sufficiently large overlap in time such that theycontain audio information of the same audio events, appropriateinformation/commands need to be exchanged between the processors beforestarting the comparison. Each sample or combination of samples withinthe time window may need to be compared with samples or combination ofsamples within the time window from the sensing microphone on the otherside. This comparison may be performed systematically or according topredefined heuristics. So far, this may be the same implementation as inthe situation described above in connection with a single cochlearimplant and a contralateral side sensing microphone only. In any case,this comparison can create a high workload for the processors, and so awork load sharing distribution between both processors may be a usefulsolution. But if the available communication rate between the processorsis relatively low, it may be better to perform the comparison in justone of the two processors. After completion of this comparison, thisprocess results in a determination of which side or sides is preferredfor further processing for the identified sound objects. The selectedtime windows may be chosen continuously (i.e. a window is defined aftereach new sample), or selected after a certain period of time (i.e at aninteger number of samples after that sample from which the previouswindows has started). This time selection may be predefined or maydepend on the previously evaluated time window samples.

Looking again at the scenario in FIG. 3 and keeping in mind that thereare communicatively coupled cochlear implants on both sides, besidesdetermining the preferred sensing microphone to use, the preferablestimulation side (left- or right-side implant) is also established. Thismay lead to a significantly improved sound localization ability of thepatient.

After the sound objects are distinguished, the processed amplitude forthe dominant implant may be further adjusted. Such amplitude adjustmentmay be performed after the processing signal is split into frequencybands as known from state-of-the-art cochlear implant systems (part ofthe stimulation signal processor 208 in FIG. 5). The amplitudeadjustment may be dependent from the frequency and the sound objectposition in order to account for effects such as head shadow and/orsquelch. Such amplitude adjustment may also be favorable in the case ofa unilateral implant with a contralateral microphone as described above.

Embodiments of the invention may be implemented in part in anyconventional computer programming language. For example, preferredembodiments may be implemented in a procedural programming language(e.g., “C”) or an object oriented programming language (e.g., “C++”,Python). Alternative embodiments of the invention may be implemented aspre-programmed hardware elements, other related components, or as acombination of hardware and software components.

Embodiments can be implemented in part as a computer program product foruse with a computer system. Such implementation may include a series ofcomputer instructions fixed either on a tangible medium, such as acomputer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk)or transmittable to a computer system, via a modem or other interfacedevice, such as a communications adapter connected to a network over amedium. The medium may be either a tangible medium (e.g., optical oranalog communications lines) or a medium implemented with wirelesstechniques (e.g., microwave, infrared or other transmission techniques).The series of computer instructions embodies all or part of thefunctionality previously described herein with respect to the system.Those skilled in the art should appreciate that such computerinstructions can be written in a number of programming languages for usewith many computer architectures or operating systems. Furthermore, suchinstructions may be stored in any memory device, such as semiconductor,magnetic, optical or other memory devices, and may be transmitted usingany communications technology, such as optical, infrared, microwave, orother transmission technologies. It is expected that such a computerprogram product may be distributed as a removable medium withaccompanying printed or electronic documentation (e.g., shrink wrappedsoftware), preloaded with a computer system (e.g., on system ROM orfixed disk), or distributed from a server or electronic bulletin boardover the network (e.g., the Internet or World Wide Web). Of course, someembodiments of the invention may be implemented as a combination of bothsoftware (e.g., a computer program product) and hardware. Still otherembodiments of the invention are implemented as entirely hardware, orentirely software (e.g., a computer program product).

Although various exemplary embodiments of the invention have beendisclosed, it should be apparent to those skilled in the art thatvarious changes and modifications can be made which will achieve some ofthe advantages of the invention without departing from the true scope ofthe invention.

What is claimed is:
 1. A sound processing arrangement for a patient witha bilateral cochlear implant system having implanted electrode arrays ineach ear, the system comprising: an ipsilateral left-side sensingmicrophone and a contralateral right-side sensing microphone, eachconfigured for sensing the sound environment surrounding the patient andgenerating corresponding microphone signals; a sound objectidentification module configured for analyzing the microphone signals toidentify a plurality of sound objects SO_(k) within the soundenvironment, the sound object identification module providing for eachk^(th) identified sound object SO_(k), two sound object subsets, SO_(kI)received from the ipsilateral left-side sensing microphone and SO_(kC)received from the contralateral right-side sensing microphone; a soundobject selection module configured to select, for each k^(th) identifiedsound object, either SO_(kI) or SO_(kC) for use as a sound object signalfor each of the plurality of sound objects; a stimulation side selectormodule configured for selecting, for each sound object signal, whichside or sides of the bilateral cochlear implant arrangement to use forstimulation; and one or more sound processors for processing each soundobject signal so as to generate a stimulation signal to the implantedelectrode arrays on the side or sides selected for the sound signalobject.
 2. An arrangement according to claim 1, wherein the one or moresound processors process the sound object signals based on adjusting aphase component of each sound object signal.
 3. An arrangement accordingto claim 1, wherein the one or more sound processors process the soundobject signals based on adjusting an amplitude component of each soundobject signal.
 4. An arrangement according to claim 1, wherein thestimulation side selector module is configured for using sound objecttime difference components in the microphone signals to select, for eachsound object, which side or sides of the bilateral cochlear implantarrangement to use for stimulation.
 5. An arrangement according to claim1, wherein the stimulation side selector module is configured for usingsound object amplitude difference components in the microphone signalsto select, for each sound object, which side or sides of the bilateralcochlear implant arrangement to use for stimulation.
 6. An arrangementaccording to claim 1, wherein the sensing microphones are located nextto the ear on each side of the patient's head.
 7. An arrangementaccording to claim 1, wherein the sensing microphones are located in theear canal on each side of the patient's head.
 8. A method of processingsound signals for a patient with a bilateral cochlear implant systemhaving implanted electrode arrays in each ear, the method comprising:sensing the sound environment surrounding the patient with anipsilateral left-side sensing microphone and a contralateral right-sidesensing microphone and generating corresponding microphone signals;analyzing the microphone signals to identify a plurality of soundobjects SO_(k) within the sound environment, and providing for eachk^(th) identified sound object SO_(k), two sound object subsets, SO_(kI)received from the ipsilateral left-side sensing microphone and SO_(kC)received from the contralateral right-side sensing microphone; selectingfor each k^(th) identified sound object, either SO_(kI) or SO_(kC) foruse as a sound object signal for each of the plurality of sound objectsSO_(k); selecting, for each sound object signal, which side or sides ofthe bilateral cochlear implant arrangement to use for stimulation; andprocessing each sound object signal so as to generate a stimulationsignal to the implanted electrode array on the side or sides selectedfor the sound signal object.
 9. A method according to claim 8, whereinprocessing the sound object signals includes adjusting a phase componentof each sound object signal.
 10. A method according to claim 8, whereinprocessing the sound object signals includes adjusting an amplitudecomponent of each sound object signal.
 11. A method according to claim8, wherein sound object time difference components in the microphonesignals are used for selecting, for each sound object, which side orsides of the bilateral cochlear implant arrangement to use forstimulation.
 12. A method according to claim 8, wherein sound objectamplitude difference components in the microphone signals are used forselecting for each sound object, which side or sides of the bilateralcochlear implant arrangement to use for stimulation.
 13. A methodaccording to claim 8, wherein the sensing microphones are located nextto the ear on each side of the patient's head.
 14. A method according toclaim 8, wherein the sensing microphones are located in the ear canal oneach side of the patient's head.
 15. A sound processing arrangement fora patient with a unilateral cochlear implant system having an implantedelectrode array in one ear, the system comprising: an ipsilateralleft-side sensing microphone and a contralateral right-side sensingmicrophone, each configured for sensing the sound environmentsurrounding the patient and generating corresponding microphone signals;a sound object identification module configured for analyzing themicrophone signals to identify a plurality of sound objects SO_(k)within the sound environment, and providing for each k^(th) identifiedsound object SO_(k), two sound object subsets, SO_(kI) received from theipsilateral left-side sensing microphone and SO_(kC) received from thecontralateral right-side sensing microphone; a sound object selectionmodule configured to select, for each k^(th) identified sound object,either SO_(kI) or SO_(kC) for use as a sound object signal for each ofthe plurality of sound objects SO_(k), wherein SO_(kI) is selected ifthe ipsilateral left-side sensing microphone is closer to the soundobject SO_(k), and wherein SO_(kC) is selected if the contralateralright-side sensing microphone is closer to the sound object SO_(k); anda sound processor configured for processing the sound object signals togenerate stimulation signals to the implanted electrode array.